operations table: Store the operation_xdr as BYTEA#497
operations table: Store the operation_xdr as BYTEA#497aditya1702 wants to merge 23 commits intohash-byteafrom
operations table: Store the operation_xdr as BYTEA#497Conversation
Part of the plan to store XDR data as raw bytes for efficiency.
Similar pattern to HashBytea but uses base64 encoding (standard for XDR) and supports variable-length data.
Changed OperationXDR field from string to XDRBytea for automatic base64/BYTEA conversion.
Cast the base64 XDR string to XDRBytea for proper database storage.
- Change UNNEST cast from text[] to bytea[] - Convert XDRBytea to raw bytes in BatchInsert - Update BatchCopy to pass raw bytes instead of pgtype.Text
Forces GraphQL to use a resolver for operationXdr to convert XDRBytea to base64 string for API consumers.
- Update test_utils.go to use types.XDRBytea - Regenerate GraphQL code with OperationXdr resolver
Returns the XDRBytea as a base64-encoded string for API consumers.
- Use parameterized queries with XDRBytea for SQL inserts - Update assertions to use .String() for comparison - Fix type casting in test data creation
- Use base64-encoded test XDR data in test_utils.go - Add testOpXDR helper functions in test files - Update all assertions to use .String() method
- Update createTestOperation to use proper base64 XDR - Update generateTestOperations to encode test data as base64 - Update operations_test.go with base64 encoding for all test XDR data - Fix assertions to compare against the stored XDRBytea values
This simplifies the XDR storage flow by storing raw bytes directly instead of encoding to base64 and then decoding. The String() method now handles base64 encoding for external representation.
Skip the intermediate base64 encoding step by using MarshalBinary() instead of MarshalBase64(). The raw bytes are now stored directly in XDRBytea.
Remove unnecessary Value() calls since XDRBytea is now []byte. Access raw bytes directly via type conversion.
Decode expected base64 XDR string to raw bytes for comparison since XDRBytea now uses []byte underlying type.
Use raw bytes directly instead of base64-encoded strings when creating test data for XDRBytea fields.
Use raw bytes directly for test XDR data instead of base64-encoding. The String() method will handle base64 encoding for assertions.
Use raw bytes directly instead of pre-encoded base64 string.
Use parameterized queries instead of raw SQL string literals for BYTEA operation_xdr column. Fix .String() assertion to compare base64 values via opXdr1.String().
Use parameterized queries instead of raw SQL string literals for BYTEA operation_xdr column in BatchGetByOperationIDs and BatchGetByStateChangeIDs tests.
Use parameterized queries instead of raw SQL string literals for BYTEA operation_xdr column in BatchGetByOperationIDs test.
Copy the byte slice from the database driver instead of referencing it directly. The pgx driver reuses its internal buffer across rows, so without copying, all scanned XDRBytea values end up pointing to the same (overwritten) buffer.
operations table: Store the operation_xdr as BYTEA
There was a problem hiding this comment.
Pull request overview
Updates the operations.operation_xdr storage to use PostgreSQL BYTEA (raw XDR bytes) while preserving the GraphQL API contract by continuing to expose operationXdr as a base64-encoded String.
Changes:
- Change
operations.operation_xdrfromTEXTtoBYTEAin the schema/migration and update DB write paths accordingly. - Introduce
types.XDRByteato scan/store raw bytes and encode to base64 when presented as a string. - Update GraphQL schema/resolvers and tests to reflect the new storage/encoding behavior.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/services/ingest_test.go | Updates ingest test fixtures to use types.XDRBytea. |
| internal/serve/graphql/schema/operation.graphqls | Forces resolver for operationXdr so GraphQL continues returning a base64 string. |
| internal/serve/graphql/resolvers/transaction_resolvers_test.go | Updates expectations for base64-encoded XDR output. |
| internal/serve/graphql/resolvers/test_utils.go | Stores operation XDR as raw bytes in test DB setup. |
| internal/serve/graphql/resolvers/queries_resolvers_test.go | Updates query resolver tests to compare base64 encoding of raw bytes. |
| internal/serve/graphql/resolvers/operation.resolvers.go | Adds OperationXdr field resolver to base64-encode XDRBytea. |
| internal/serve/graphql/resolvers/account_resolvers_test.go | Updates account resolver tests for base64-encoded XDR output. |
| internal/serve/graphql/generated/generated.go | Regenerates gqlgen output to wire the forced resolver. |
| internal/indexer/types/types.go | Adds XDRBytea type and updates types.Operation.OperationXDR to use it. |
| internal/indexer/processors/utils_test.go | Updates ConvertOperation test to compare raw bytes vs base64 string. |
| internal/indexer/processors/utils.go | Stores raw XDR bytes via MarshalBinary() instead of base64 strings. |
| internal/db/migrations/2025-06-10.3-operations.sql | Changes operation_xdr column type to BYTEA (but see migration concern). |
| internal/data/transactions_test.go | Updates test inserts to pass XDRBytea for operation_xdr. |
| internal/data/operations_test.go | Updates operation model tests for bytea storage and base64 string comparisons. |
| internal/data/operations.go | Updates batch insert/copy paths to send operation_xdr as bytea. |
| internal/data/accounts_test.go | Updates account model tests inserting operations to use XDRBytea. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ), | ||
| operation_xdr TEXT, | ||
| operation_xdr BYTEA, | ||
| result_code TEXT NOT NULL, |
There was a problem hiding this comment.
Changing the column type in this existing migration file won’t update already-migrated databases, so production/dev DBs that have operation_xdr as TEXT will keep that type and will likely break once the app starts inserting BYTEA. Add a new forward migration that ALTER TABLE operations ALTER COLUMN operation_xdr TYPE BYTEA USING decode(operation_xdr, 'base64') (or equivalent), and avoid rewriting already-shipped migrations.
| id: Int64! | ||
| operationType: OperationType! | ||
| operationXdr: String! | ||
| operationXdr: String! @goField(forceResolver: true) |
There was a problem hiding this comment.
The PR description still contains TODO placeholders (What/Why/Issue/Checklist). Please fill these in so reviewers understand the rationale, rollout/migration plan, and any breaking-change considerations for the operation_xdr storage change.
What
[TODO: Short statement about what is changing.]
Why
[TODO: Why this change is being made. Include any context required to understand the why.]
Known limitations
[TODO or N/A]
Issue that this PR addresses
[TODO: Attach the link to the GitHub issue or task. Include the priority of the task here in addition to the link.]
Checklist
PR Structure
allif the changes are broad or impact many packages.Thoroughness
Release